Developments in Probabilistic Modelling with Neural Networks|ensemble Learning 1 Ensemble Learning by Free Energy Minimization

Ensemble learning by variational free energy minimization is a framework for statistical inference in which an ensemble of parameter vectors is optimized rather than a single parameter vector. The ensemble approximates the posterior probability distribution of the parameters. In this paper I give a review of ensemble learning using a simple example. A new tool has recently been introduced into the eld of neural networks. In traditional approaches to model tting, a single parameter vector w is optimized by, say, maximum likelihood or penalized maximum likelihood; in the Bayesian interpretation, these optimized parameters are viewed as deening the mode of a posterior probability distribution P (wjD; H) (given data D and model assumptions H). The new concept introduced by Hinton and van Camp (1993) is to work in terms of an approximating ensemble Q(w;), that is, a probability distribution over the parameters, and optimize the ensemble (by varying its own parameters) so that it approximates the posterior distribution of the parameters P (wjD; H) well. The objective function chosen to measure the quality of the approximation is a variational free energy, 1